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Annotated  Bibli  j^ra^Z_of_Pa£ers_Prepared_f or_this^Proj ect 

Carbonall,  J.  R.  and  Collins,  A.  M.  Natural  semantics  in 
artlf icial  intelligence.  In  Proceedings  of  the  Third 
^l£g£?IitL°rcal_Joint_Conferencej^_Artificiaf  Intelligence, 
Stanford  University,  1973  , 344-351.  ReprintedTn'the 

^er^an  Journal  of  Computational  Linguistics,  1,  Mfc  3,  197  4. 
(Partial  support) 

This  paper  discusses  human  semantic  knowledge  and 
processing  in  terms  of  the  SCHOLAR  system,  in  one 
major  section  we  discuss  the  imprecision,  the  incom- 
pleteness, the  open-endedness,  and  the  uncertainty  of 
people's  knowledge.  In  the  other  major  section 
we  discuss  strategies  people  use  to  make  different 
types  of  deductive,  negative,  and  functional  inferences, 
and  the  way  uncertainties  combine  in  these  inferences. 
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Collins,  A.  M.,  Passaf iume,  J.  J. , Gould,  L. , and  Carbonell,  J.  G. 
Improving  interactive  capabilities  in  computer-assisted 
instruction.  BBN  Report  No.  2631,  1973. 


This  report  describes  the  development  of  interactive 
capabilities  in  the  SCHOLAR  CAI  system  centering  in  three 
main  areas:  (1)  implementation  of  two  presentation 
strategies  in  SCHOLAR  (Tutorial  mode  and  Block-Test  mode) 
and  a comparative  evaluation  of  these  two  modes  usiug 
high-school  students  as  subjects;  (2)  initial  study  based 
on  analysis  of  tutorial  dialogues  of  how  to  teach 


1 


procedural  knowledge  interactively  within  SCHOLAR,  and 
(3)  addition  of  a module  for  teaching  geography  using  the 
map  display  and  related  question-answering  facilities 
recently  added  tc  SCHOLAR. 


Collins,  A.  M. , Warnock,  E.  H.,  and  Passafiume,  J.  J.  Analysis 
and  synthesis  of  tutorial  dialogues.  In  G.  Bower  (Ed.), 

The  psychology  of  learning  and  not i nation,  Vol . 9 . 

New  York:  Academic  Press,  1975. 


In  this  paper  we  attempt  to  analyze  the  strategies  by 
which  tutors  adapt  their  teaching  to  individual  students, 
so  that  we  can  synthesize  these  strategies  in  the 
SCHOLAR  CAI  system.  To  find  out  what  strategies 
tutors  use,  we  tape-recorded  dialogues  between  various 
tutors  and  students  on  the  topic  of  South  American 
geography.  Because  GCHOLAR  is  a well-defined  program, 
it  is  possible  to  analyze  such  ill-defined  naturalistic 
data  in  precise  terms  with  respect  to  the  structure 
and  processing  of  information  in  SCHOLAR.  We  analyzed 
the  dialogues  concentrating  on  one  aspect  at  a time. 

Based  on  our  analyses,  we  propose  in  this  paper  several 
hypotheses  about  how  the  tutor  relates  his  teaching  to 
the  individual  student.  We  show  how  in  modified  form 
we  have  implemented  some  of  these  strategies  in  SCHOLAR. 

We  further  argue  that  the  analytical  method  employed  here 
could  be  extended  to  a wide  range  of  conversational 
situations.  This  method  (Dialogue  Analysis)  would  permit 
psychologists  to  study  questions  about  the  interactive 
aspects  of  human  processing  that  cannot  even  be  considered 
with  traditional  laboratory  methods. 
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Collins,  A.  M.  Comparison  of  two  teaching  strategies  in  computer- 
assisted  instruction.  BBN  Report  No.  2885,  1974.  Submitted 
to  Instructional  Science. 

Three  experiments  were  run  using  the  SCHOLAR  CAI  system 
to  teach  geography  to  high-school  students.  The  experi- 
ments compared  a method  of  teaching  derived  from  analysis 
of  human  tutors  (Tu corial  mode)  vs.  a method  derived 
from  programmed  instruction  (Block-Test  mode) . In  the 
thr66  experiments.  Block— Test  mode  was  systematically 
converged  toward  Tutorial  mode  in  order  to  pinpoint  what 
aspects  of  teaching  strategy  affected  students'  learning. 
Tutorial  mode  was  significantly  more  effective  in  the 
first  two  exp ■ i iments , and  nonsignif icantly  in  the  third. 
The  results  Indicated  that  the  major  factor  affecting 
students'  learning  was  the  strategy  that  tutors  use  of 
reviewing  the  material  in  greater  depth  on  a second  pass. 
Allowing  the  students  to  ask  questions,  and  the  tutorial 
strategy  for  relating  new  material  to  the  students' 
previous  knowledge  contributed  only  a small  amount  to 
the  differences  found  in  the  first  two  experiments. 


Grignetti,  M.  C.,  Hausmann,  C.,  and  Gould,  L.  An  "intelligent" 
on-line  assistant  and  tutor— NLS-SCHOLAR.  In  Proceedings  of 
the  National  Computer  Conference,  San  Diego,  California,  1975, 
775-781.  (Partial  support) 


NLS-SCHOLAR  is  a prototype  system  that  uses  Artificial 
Intelligence  techniques  to  teach  computer-naive  people 
how  to  use  a powerful  and  complex  editor.  It  represents 


a new  kind  of  Computer  Assisted  Instruction  (CAI)  system 
that  integrates  systematic  teaching  with  actual  practice, 
i.e.,  one  which  can  keep  the  user  under  tutorial  super- 
vision while  allowing  him  to  try  out  what  he  learns  on 
the  system  he  is  learning  about. 

NLS-SCHOLAR  can  also  be  used  as  an  on-line  help  system 
outside  the  tutorial  environment,  in  the  course  of  a 
user's  actual  work.  This  capability  of  combining  on-line 

assistance  with  training  is  an  extension  of  the  traditional 
notion  of  CAI. 

The  techniques  used  in  NLS-SCHOLAR  are  general  and  can 
be  applied  to  a wide  variety  of  computer-related  activities 


Collins,  A.  M.,  WarnocK,  E.  H.,  Aiello,  N.,  and  Miller,  M.  W. 
Reasoning  from  incomplete  knowledge.  In  D.  G.  Bobrow  and 
A.  M.  Collins  (Eds.),  Representation  and  understand . 

New  York:  Academic  Press,  1975. 

The  paper  describes  how  people  use  a variety  of  plausible, 
but  uncertain,  inferences  to  answer  questions  about  which 
their  knowledge  is  incomplete.  This  kind  of  reasoning  is 
described  in  terms  of  how  it  is  being  implemented  in  the 
SCHOLAR  CAI  system.  The  paper  also  shows  how  people  can 
be  taught  to  reason  in  this  way,  using  a Socratic  tutorial 
method  implemented  in  a system  like  SCHOLAR. 
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E"ucation  “0  understanding.  i„  Klahr  (Ed.) 
Cognji^aj^st^ej:^.  Hillsdale,  N.  j. , Erlbaujn 
Associates , 1975. 

This  Chapter  comments  on  chapters  by  Just  and  Carpenter 

y imon  and  Hayes  on  teaching  understanding  skills 
The  chapter  argues  that  the  tost  important  aspect  of 

understanding  is  how  people  use  their  knowledge  about 
thiG  wor id  to  f i 11  in  « c , 

Thus  th  „ information  the  text  assumes. 

re  can  be  no  easy  way  to  educate  people  to 
understand,  because  they  need  to  be  taught  both  a huge 
amount  o,  world  knowledge  and  the  understanding  ski  Us 
to  use  that  knowledge  effectively  in  reading. 

COlB1'"3;  A‘  an<i  Grl^etti'  »•  C-  Intelligent  CAI. 

ReP°rt  N°-  3181'  l975‘  »>.  submit  ed  to  Science. 

This  paper  describes  the  capabilities  now  available  for 
riding  intelligent,  tutorial  CAI  systems  as  exemplify, 

inClUdi"9  TUtOr-SCH0“R'  Map-SCHOLAR. 
NLS  SCHOLAR  and  SOPHIE.  The  systems  illustrate  how  a 

y o sophisticated  techniques  can  be  used  for 
tutoring  different  kinds  of  knowledge  by  carrying  on 
alogues  in  natural  language.  The  systems  have  been 
developed  to  explore  how  to  provide  each  student  with 
his  own  personal,  expert  tutor. 
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Collins , A. , Pew,  R.  W. , and  Adams,  M.  The  effectiveness 
of  an  interactive  map  display  in  tutoring  geography.  In 
preparation. 

This  paper  will  describe  the  Map-SCHOLAR  system  and  an 
experiment  that  compares  how  well  students  learn  in 
Tutorial  Mode,  using  (a)  the  interactive  map  display 
of  Map-SCHOLAR  (b)  a static  labeled  map,  and  (c)  an 
unlabeled  map.  The  paper  will  also  show  how  a new 
method  called  backtrace  analysis  can  be  used  to 
pinpoint  the  effectiveness  of  different  aspects  of 
the  tutoring  strategy  and  the  map  system  used  in  the 
experiment. 
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INTRODUCTION 

If  computerized  instruction  is  ever  to  have  a large  impact  on 
education,  computer-assisted  instruction  (CAI)  systems  must  have  the 
flexibility  and  skill  of  a human  teacher.  In  developing  the  SCHOLAR 
CAI  system  Carbonell  (1)  took  a first  step  toward  an  intelligent 
tutorial  CAI  system.  In  SCHOLAR,  knowledge  was  not  stored  as  text, 
but  in  an  interrelated  network  of  facts  and  concepts,  so  that  the 
knowledge  could  be  used  in  a variety  of  ways.  In  short  the  attempt 
was  to  structure  information  like  a human  knowledge,  so  that  the 
program  could  use  its  knowledge  as  flexibly  as  a human  tutor  does. 

In  this  paper  we  will  discuss  the  structure  of  the  SCHOLAR 
system,  some  of  the  ways  that  the  potential  for  intelligent  CAI  has 
been  realized  in  current  systems,  and  finally  wnat  is  possible  in 
the  near  future  toward  building  intelligent  tutorial  systems. 

The  Context  of  SCHOLAR  in  CAI 

Prior  to  Carbonell 's  SCHOLAR  program,  CAI  had  proceeded  along 
several  lines.  Bryan  (2)  distinguished  three  broad  categories.  In 
the  first,  ad-hoc  CAI,  the  student  is  given  full  control  of  the 
computer  with  a simple  programming  language  and  perhaps  a series  of 
tasks  to  perform.  LOGO  (3)  provides  one  of  the  most  interesting 
educational  environments  of  this  kind  and  indeed  children  learn  some 
important  cognitive  skills  in  working  with  LOGO.  The  second 
category  is  games  and  simulation,  where  the  student  learns 
indirectly  while  participating  in  the  game  or  simulation.  The  Plato 
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system's  "How  the  west  was  won?"  (4)  is  an  excellent  example  of  such 
a system  where  children  learn  the  arithmetic  operations  in  playing  a 
variant  of  "Chutes  arJ  Ladders."  Both  these  forms  of  CAI  are  highly 
interactive,  but  they  are  limited  as  teaching  methods  to  certain 
kinds  of  knowledge. 

The  third  category  Bryan  called  controlled  learning.  Most 
programs  in  this  category  specify  the  possible  sequences  through  a 
program,  where  different  branches  are  taken  depending  on  the 
student's  responses  to  questions  or  problems.  The  sequence  a 
student  follows  is  usually  deterministic,  with  a branch  for  each 
anticipated  class  of  responses  by  the  student  (sometimes  based  on  a 
keyword  he  might  give).  Some  ingenious  programs  can  be  written  in 
this  way,  such  as  the  Socratic  system  (5)  or  the  chemistry  programs 
in  the  Plato  System  (6),  but  there  are  some  inherent  limitations  to 
this  approach.  The  student  can  not  use  natural  language  in  his 
responses,  and  cannot  ask  any  but  specifically  anticipated 
questions.  The  teacher  has  a considerable  burden  in  the  preparation 
of  questions,  answers,  keywords,  and  branchings.  From  a system's 
point  of  view,  the  system  has  no  real  initiative  or  decision  power 
of  its  own,  nor  any  knowledge  tnat  is  available  other  than  at  fixed 
points  in  the  sequence. 


The  rigidity  of  this  latter  approach  led  to  the  development  of 
"generative  CAI"  (7).  The  first  CAI  programs  were  based  on 
mathematics  and  other  well-defined  subjects,  where  problems  could  be 
generated  and  answers  evaluated  by  "algorithmic  manipulation".  This 
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was  a major  step  toward  representing  a program  s knowledge  so  that 
it  could  be  used  in  a variety  of  ways.  But  the  knowledge  was 
limited  to  algorithmic  domains. 

Carbonell's  contribution  was  to  apply  the  idea  of  generative 
CAI  to  ill -defined  verbal  knowledge,  in  particular  the  domain  of 
geography.  To  do  this  he  used  a semantic  network  (8)  for 
representing  knowledge  about  the  world.  In  a semantic  network, 
information  is  scored  about  concepts  under  different  entries,  much 
as  in  an  encyclopedia,  but  according  to  a well-defined  format. 
Every  concept  used  to  describe  a given  concept  can  itself  be 
described  elsewhere  in  the  network.  Thus  the  program  can  know  about 
e’ cry  concept  it  uses. 

When  information  is  represented  in  the  general  format  a 
semantic  network,  then  the  structure  of  the  network  can  be  used  by 
different  procedures  to  formulate  questions  to  the  student,  evaluate 
his  answers,  answer  his  questions,  make  inferences  and  computations, 
select  new  topics  for  discussion,  etc.  Freeing  the  information  from 
the  specific  use  it  is  to  be  put  makes  it  available  for  use  in  a 
variety  of  ways.  In  this  paper  we  will  try  to  show  some  of  the 
power  for  tutorial  interaction  that  can  be  obtained  by  such  an 
approach  to  CAI . 

The  Original  SCHOLAR  Program 

Figure  1 presents  a short  protocol  with  SCHOLAR , showing  some 
of  the  original  system's  capabilities.  SCHOLAR  selected  topics  on  a 
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WHAT  ARE  THE  PRINCIPAL  CITIES  OF  BRAZIL? 

•RIO  AND  MAYBE  BUENOS  AIRES  OR  SAN  PAULO* 

RIO  DE  JANEIRO  AND  SAO  PAULO  ARE  CORRECT.  YOU  MISSPELLED 
SAO  PAULO.  BUENOS  AIRES  IS  IN  ARGENTINA,  NOT  IN  BRAZIL. 
THE  OTHER  PRINCIPAL  CITIES  OF  BRAZIL  ARE  BRASILIA,  PORTO 
ALEGRE,  SALVADOR,  RECIFE,  SANTOS,  MANAOS,  AND  BELEM. 

TRUE  OR  FALSE?  THE  LANGUAGE  IN  BRAZIL  IS  FRENCH. 


*Q/A* 

DO  YOU  WANT  TO  CHANGE  THE  GENERAL  MODE  OF  THE  INTERACTION  TO 
QUESTION-ANSWERING  MODE?  PLEASE  ANSWER  Y OR  N. 

*Y* 

OK,  MR.  JONES.  LET'S  GO  AHEAD. 

•WHERE  IS  I RASILIA?* 

BRASILIA  IS  IN  CENTRAL  BRAZIL.  IT  I§  IN  THE  BRAZILIAN 
HIGHLANDS. 

•TELL  ME  MORE  ABOUT  THE  LOCATION  OF  BRASILIA?* 

THE  LATITUDE  OF  BRASILIA  IS  -16.  THE  LONGITUDE  IS  -48. 

•WHAT  IS  LATITUDE?* 

A LATITUDE  IS  AN  ANGULAR  DISTANCE  FROM  THE  EQUATOR.  IT  CAN 
BE  CONSIDERED  A PART  OF  A LOCATION. 

•WHAT  IS  THE  MOST  IMPORTANT  DIFFERENCE  BETWEEN  BRASILIA  AND 
RIO* 

THE  PRIMARY  DIFFERENCE  IS:  BRASILIA  IS  IN  CENTRAL  BRAZIL. 
BUT  RIO  DE  JANEIRO  IS  IN  SOUTHEASTERN  BRAZIL. 

•IS  THE  CLIMATE  OF  SAO  PAULO  TROPICAL* 

NO.  THE  CLIMATE  OF  SAO  PAULO  IS  TEMPERATE. 


Fiq.  1 Dialogue  with  the  original  SCHOLAR 
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random  basis,  weighted  for  importance.  When  it  printed  an  asterisk, 
the  student  could  respond  to  the  question  by  SCHOLAR,  direct  a 
question  to  SCHOLAR,  or  issue  a command  such  as  halting  or  changing 
the  mode  of  interaction.  The  student  returned  control  to  SCHOLAR  by 
typing  an  asterisk  and  carriage  return. 

SCHOLAR'S  questions  were  constructed  from  the  information  in 
the  semantic  network,  and  were  not  prestored.  The  student's  answers 
were  evaluated  by  different  procedures  with  respect  to  the 
information  in  the  network,  deciding  whether  the  particular  answer 
was  correct,  wrong,  partly  correct,  or  approximately  correct. 
Student's  questions  were  analyzed  syntactically  to  determine  what 
information  was  sought,  and  the  semantic  network  searched  to  find 
the  information.  Different  computations  and  inferences  were 
applied,  if  the  information  was  not  stored  directly.  All  these 
operations  were  carried  out  by  procedures  which  operated 
independently  of  the  specific  information  that  was  involved. 


In  Fig.  1 the  questions  by  SCHOLAR  illustrate  different  kinds 
of  questions  that  could  be  generated.  After  the  second  question  by 
SCHOLAR,  the  student  changed  the  mode  of  interaction  from 
mixed-initiative  mode,  where  SCHOLAR  asked  him  questions,  to 
question-answering  mode,  where  SCHOLAR  waited  for  questions  from  the 
student.  Other  modes  described  below  have  since  been  added  to 


SCHOLAR.  Because  the  student  can  control  the  way  he  interacts  with 
the  system,  he  can  choose  the  mode  of  interaction  that  he  finds  most 


effective.  This  is  one  of  the  important  ways  such  a system 
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personalizes  instruction. 

The  student  then  asked  a series  of  questions  to  clarify  and 
extend  the  information  given  to  him  about  the  cities  of  Brazil. 
When  the  student  wants  more  information  about  something  such  as 
Brazilia,  he  can  ask  specifically  what  he  wants  to  know.  When  he 
doesn't  understand  a word,  such  as  latitude,  he  can  have  it 
explained.  In  this  way  the  knowledge  taught  can  be  geared  to  the 
individual  student's  background,  so  as  not  to  repeat  what  he  already 
knows  or  go  over  his  head.  This  is  important  to  maintaining  a 
student's  motivation  to  learn. 


I 


The  two  questions  about  the  location  of  Brasilia  illustrate  how 
a tutorial  system  can  avoid  overloading  the  student  with  too  much 
information  at  one  time.  Each  piece  of  information  in  the  network 
is  tagged  to  indicate  its  relative  importance.  The  program  gives 
only  the  most  important  information  at  any  time,  but  the  student  can 
always  ask  for  more  information  if  he  wants  it. 

The  questions  about  the  most  important  difference  between 
Brasilia  and  Rio  and  about  the  climate  of  Sao  Paulo  illustrate  the 
ability  to  use  a semantic  network  to  make  appropriate  computations 
and  inferences.  In  the  first  case  there  is  a procedure  for 
comparing  two  things  to  find  their  similarities  and/or  differences. 
Each  property  of  the  two  things  is  compared  in  the  order  of 
importance.  Here  the  most  important  property  on  which  the  two 
things  differ  is  found,  and  given  as  an  answer  to  the  student.  The 
second  case  illustrates  a combination  of  two  inferences,  a deduction 
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and  a contradiction.  Nothing  about  climate  is  stored  with  Sao 
Paulo,  but  Sao  Paulo  is  in  the  Brazilian  Highlands  which  has  a 
temperate  climate.  By  comparing  tropical  and  temperate,  SCHOLAR 
finds  there  is  a contradiction  and  concludes  the  answer  is  "no". 
There  are  a large  number  of  such  inferential  strategies  that  humans 
use,  and  only  some  of  the  more  common  ones  have  been  implemented  in 
SCHOLAR.  But  information  in  SCHOLAR  is  structured  in  such  a way 
that  it  is  possible  to  specify  content-independent  procedures  to 
carry  out  different  inferences. 

This  summarizes  the  major  contributions  of  the  original  SCHOLAR 
system.  There  were  also  several  severe  limitations  to  the  original 
SCHOLAR.  First,  the  information  in  the  program  was  restricted  to 
static,  verbal  facts  about  geography,  which  are  not  very  interesting 
in  themselves.  Second,  the  program  was  quite  restricted  in  its 
ability  to  understand  student  answers  and  questions  because  of  its 
limited  language  processing  capability.  Third,  and  perhaps  most 
important  there  was  no  teaching  strategy;  the  program  merely 
generated  questions  randomly  or  answered  student  questions.  We  will 
try  to  show  how  later  systems  have  overcome  some  of  the  limitations 
of  the  original  SCHOLAR  and  at  the  same  time  exploited  further  its 
potential  for  tutorial  interaction. 
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TEACHING  STRATEGY 
Analyst?  Of  Human  Tutoring 

In  the  original  SCHOLAR  there  was  no  teaching  strategy,  but 
SCHOLAR'S  structure  made  it  possible  to  model  the  way  human  tutors 
interact  with  students.  By  collecting  tape  reco-.-dings  of  different 
tutors  teaching  the  same  kind  of  information  as  SCHOLAR,  it  was 
possible  to  analyze  how  tutors  adapt  their  teaching  to  the 
individual  student  (9).  There  were  four  crucial  aspects  of  their 
tutoring  strategy,  that  were  subsequently  modelled  in  SCHOLAR.  They 
were:  (a)  the  way  tutors  select  topics,  (b)  the  way  they  interweave 
questions  and  presentation,  (c)  their  reviewing,  and  (d)  their  error 
correction  strategy. 

The  topic  selection  strategy  used  by  tutors  produces  a 
structure  of  topics  and  subtopics  like  an  outline  for  a course.  For 
example,  the  tutor  might  start  off  with  a question  like  "Do  you  know 
any  geographical  features  of  South  America?"  If  the  student  gives 
Cape  Horn,  for  example,  then  the  tutor  would  discuss  Cape  Horn  for  a 
while,  including  perhaps  the  Straits  of  Magellan  as  a subtopic. 
After  covering  the  most  important  information  about  Cape  Horn,  the 
tutor  would  then  ask  about  other  geographical  features,  like  the 
Amazon  or  the  Andes.  Each  of  these  would  be  discussed  briefly  until 
the  major  geographical  features  are  covered,  at  which  point  the 
tutor  would  pick  a new  topic  such  as  regions  or  countries.  Thus, 
the  topics  and  subtopics  form  a nested  outline  structure,  with  the 
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tutor  probing  a little  way  into  each  subtopic,  and  then  pepping  up 
to  the  previous  topic  when  the  important  information  is  exhausted. 
The  better  the  tutor,  the  more  structure  there  is  to  the  discussion. 

The  way  the  tutor  interweaves  questioning  and  presentation  is 
the  essence  of  how  the  tutor  relates  his  teaching  to  the  individual 
student.  The  dialogues  showed  that  the  tutors'  questions  occur  at 
the  top-level  and  beginning  topics  in  the  outline.  This  is  because 
the  tutor  starts  out  asking  questions  to  find  out  what  the  student 
already  knows,  and  then  presents  new  material  that  is  related  to  the 
student's  previous  knowledge.  The  object  seems  to  be  to  tie  as  much 
information  as  the  student  can  assimilate  into  the  structure  of  his 
previous  knowledge  (10). 

Another  important  aspect  of  the  tutorial  strategy  is  reviewing. 
In  the  dialogues  the  better  tutors  went  over  the  material  on  a 
second  pass,  asking  about  things  the  student  didn't  know  the  first 
time  through,  and  adding  more  detail  to  the  structure  of  information 
built  up  on  the  first  pass.  The  tutorial  method  as  a whole 
reflected  a strategy  Norman  (10)  refers  to  as  "web  teaching",  where 
the  teacher  first  tries  to  establish  a framework  of  basic  knowledge 
and  then  fills  in  more  aid  more  detail  on  subsequent  passes,  much 
like  a spider  spinning  a web. 

The  fourth  aspect  of  the  dialogues  important  to  individualizing 
instruction  is  the  way  tutors  correct  student  errors.  When  students 
make  a confusion  between  two  concepts,  the  better  tutors  try  to 
provide  distinguishing  properties  between  the  concepts  for  the 
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student.  For  example,  when  one  student  confused  Ecuador  and 
Colombia,  the  tutor  pointed  out  that  Ecuador  is  a much  smaller 
country  and  that  Colombia  is  connected  to  Panama.  By  providing 
distinguishing  characteristics,  the  tutor  is  giving  the  individual 
the  most  relevant  information  for  remembering  v he  distinction  m the 

future . 


Tutorial  Mode  in  SCHOLAR 

These  four  aspects  of  the  human  tutoring  strategy  were 
developed  in  a mode  called  Tutor-SCHOLAR  (9).  Like  the  human  tutor, 
Tutor-SCHOLAR  selects  topics  in  order  of  importance,  and  goes  into 
depth  on  those  topics  the  student  knows  something  about.  It  starts 
out  asking  questions  always  probing  deeper  until  the  student  doesn  t 
know  an  answer  or  the  time  is  used  up  for  that  topic.  When  the 
student  misses  a question,  it  presents  a little  related  information 
for  the  student  to  assimilate  before  going  onto  the  next  uopic.  It 
allocates  its  time  between  a first  pass  and  a review  pass.  On  the 
review  pass  it  skips  over  what  the  student  knew  earlier.  However, 
it  asks  about  everything  the  student  missed,  or  that  was  presented 
earlie”.  Anything  the  student  remembers  from  the  earlier  pass  leads 
the  discussion  into  more  depth  on  the  review  pass. 

When  the  student  suggests  an  answer  that  differs  from  what  is 
stored,  Tutor-SCHOLAR  tries  to  find  a basis  for  rejecting  the 
student's  answer.  This  involves  an  inferential  comparison  between 
what  the  student  says  and  what  is  stored.  If  it  can  find  a 
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contradiction,  the  program  points  out  the  basis  for  rejecting  the 
student's  answer  plus  anything  it  knows  about  the  relation  of  the 
student's  answer  to  the  object  in  question.  In  this  way  it  tries  to 

give  the  student  a basis  for  rememtering  the  distinction  in  the 
future . 

Figure  2 shows  a transcript  between  Tutor-SCHOLAR  and  a naive 
student  who  was  testing  out  the  system.  It  shows  the  kinds  of 
mistakes  SCHOLAR  makes  in  language  understanding.  But  it  also  shows 

the  interactive  flavor  of  SCHOLAR  with  a tutoring  strategy  directing 
the  conversation. 


Because  the  information  in  SCHOLAR  is  distinct  from  the 
procedures  used  for  teaching  that  information,  it  is  possible  to 
vary  the  teaching  strategy  while  holding  the  information  constant. 
This  makes  it  possible  to  compare  different  teaching  methods  in  a 
systematic  manner.  To  evaluate  the  effectiveness  of  Tutor-SCHOLAR 
we  compared  it  to  a teaching  strategy  derived  from  programmed 
instruction,  which  was  implemented  in  a mode  called  Block-SCHOLAR . 
Block-SCHOLAR  sexects  topics  in  the  same  order  as  Tutor-SCHOLAR,  but 
unlike  Tutor-SCHOLAR  i *■  first  presents  material  and  then  asks 
questions  about  the  material  presented. 

Three  experiments  were  run  to  compare  Tutor-SCHOLAR  and 
Block-SCHOLAR  (11).  The  comparison  was  made  by  teaching  students 
the  same  domain  of  information  with  the  two  versions  of  SCHOLAR,  and 
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LET'S  BEGIN  BY  TALKING  ABOUT  THE  LOCATION  OF  BRAZIL.  TRUE 
OR  FALSE?  BRAZIL  IS  IN  NORTHEASTERN  SOUTH  AMERICA. 

•THAT  IS  FALSE* 

NO.  BRAZIL  IS  IN  NORTHEASTERN  SOUTH  AMERICA. 

THE  EASTERN  COAST  OF  BRAZIL  IS  THE  ATLANTIC.  THE  WESTERN 
BORDERING  COUNTRIES  ARE  PERU  AND  COLOMBIA.  THE  SOUTHERN 
BORDERING  COUNTRIES  ARE  URUGUAY  AND  ARGENTINA. 

Because  the  student  missed  the  question , SCHOLAR  presents 
some  related  information  about  the  location  of  Brazil, 
and  then  goes  on  to  the  next  topic. 


NEXT  WE'RE  GOING  TO  DISCUSS  THE  TOPOGRAPHY  OF  BRAZIL.  THE 
PRINCIPAL  RIVERS  OF  BRAZIL  ARE . 

•THE  AMAZON  AND  THE  ORINOCO.* 

THE  AMAZON  IS  CORRECT.  THE  ORINOCO  IS  IN  VENEZUELA  AND 
COLOMBIA,  NOT  IN  BRAZIL.  THE  OTHER  PRINCIPAL  RIVERS  OF 
BRAZIL  ARE  THE  TOCANTINS,  THE  SAO  FRANCISCO,  THE  RIO  NEGRO , 
AND  THE  MADEIRA. 

THE  SAO  FRANCISCO  IS  IN  EASTERN  BRAZIL.  IT  IS  IN  THE 
BRAZILIAN  HIGHLANDS.  THE  RIO  NEGRO  IS  IN  THE  AMAZON  JUNGLE. 
IT  IS  IN  NORTHWESTERN  BRAZIL. 


Because  the  student  knew  about  the  Amazon , SCHOLAR  takes 
it  up  as  a topic  for  a while. 


CORRECT  OR  INCORRECT?  THE  AMAZON  IS  NAVIGABLE. 
•POSSIBLY  YES* 

NO.  THE  AMAZON  IS  NAVIGABLE. 


SCHOLAR  misunderstood  the  "possibly " and  treated  the 
answer  as  wrong. 


THE  AMAZON  IS  IN  THE  AMAZON  JUNGLE.  IT  IS  IN  BRAZIL  AND 
PERU.  THE  PRINCIPAL  TRIBUTARIES  ARE  THE  RIO  NEGRO,  THE 
MADEIRA,  THE  TOCANTINS,  THE  TAPAJOZ,  THE  PURUS,  AND  THE 
JURUA. 

WHAT  IS  THE  LENGTH  OF  THE  AMAZON? 


Fig.  2 dialogue  with  Tutor-SCFOLAF 
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•IS  THE  AMAZON  A LAKE?* 

I DON'T  KNOW  IF  THE  AMAZON  IS  A LAKE,  BUT  IT  IS  A RIVER. 


SCHOLAR  found  that  both  lakes  and  rivers  were  water , 
so  it  couldn't  reject  the  idea. 

NOW  YOU  CAN  EITHER  ANSWER  MY .QUESTION  OR  ASK  A NEW  ONE. 

•WHAT  IS  THE  QUESTION?* 

WHAT  IS  THE  LENGTH  OF  THE  AMAZON? 

•5000  MILES* 

NO.  THE  LENGTH  OF  THE  AMAZON  IS  4,000  MILES. 


Having  exhausted  its  allotted  time  to  rivers,  SCHOLAR 
went  onto  the  next  subtopic  under  topogi'aphy  of  Brazil. 


THE  PRINCIPAL  MOUNTAINS  OF  BRAZIL  ARE  . 

•I  AM  NOT  SURE,  ARE  THE  ANDES  IN  BRAZIL?* 

IN  WHAT  YOU  SAY,  I DON'T  RECOGNIZE  THE  WORDS  "I",  "AM"  AND 
"SURE".  TRY  AGAIN. 


•ARE  THE  ANDES  IN  BRAZIL?* 

NO.  THE  ANDES  ARE  NOT  IN  BRAZIL,  BUT  IN  VENEZUELA, 
COLOMBIA,  ECUADOR,  PERU,  BOLIVIA,  CHILE,  AND  ARGENTINA. 

FEEL  FREE  TO  ANSWER  THE  QUESTION  YOU  DIDN'T  „NSWER  BEFORE. 


•BRAZIL  HAS  NO  MOUNTAINS.* 

NO.  I DON'T  KNOW  WHAT  YOU  MEAN  BY  NO.  THE  CORRECT  ANSWFR 
IS  THE  BRAZILIAN  HIGHLANDS. 


SCHOLAR  didn't  understand  .the  answer,  but  proceeded  iuite 
reasonably . 


THE  BRAZILIAN  HIGHLANDS  ARE  IN  SOUTHEASTERN  BRAZIL.  THEY 
ARE  NEAR  THE  ATLANTIC. 

WHAT  IS  THE  PRINCIPAL  PLATEAU  OF  BRAZIL? 

The  dialogue  continued  discussing  different  aspects  of 
Brazil,  but  we  will  break  here  to  the  beginning  of  the 
second  pass. 


► to 
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NOW  WE  WILL  GO  OVER  THE  MATERIAL  AGAIN.  LET'S  BEGIN  BY 
TALKING  ABOUT  THE  LOCATION  OF  BRAZIL. 

PLEASE  INDICATE  IF  ThF.  FOLLOWING  STATEMENT  IS  CORRECT  OR 
INCORRECT:  BRAZIL  IS  IN  NORTHEASTERN  SOUTH  AMERICA. 

•TRUE* 

THAT'S  RIGHT.  BRAZIL  IS  IN  NORTHEASTERN  SOUTH  AMERICA. 

Because  on  this  pass  the  student  uas  correct,  SCHOLAR 
asked  questions  about  the  information  that  ues  presented 
the  first  pass. 


THE  EASTERN  COAST  OF  BRAZIL  IS  . 

•ATLANTIC  SEA.* 

VERY  GOOD.  THE  ATLANTIC  IS  CORRECT. 

THE  WESTERN  BORDERING  COUNTRIES  OF  BRAZIL  ARE . 

The  dialogue  continued  through  the  second  pass. 


Fiq.  2 Pialoquo  with  Tutor-FCHOLAP  (cont) 
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measuring  their  learning  by  comparing  pre-test  and  pose-test  scores 
on  the  material  covered.  In  the  three  experiments,  Block-SCHOLAR 
was  systematically  converged  toward  Tutor-SCHOLAR  in  order  to 
pinpoint  what  aspects  of  teaching  strategy  affected  student 's 
learning.  Tutor-SCHOLAR  was  significantly  more  effective  in  the 
first  two  experiments,  and  nonsignificantly  in  the  third.  The 
results  indicated  that  the  major  factor  affecting  student's  learning 
was  the  strategy  that  tutors  use  of  reviewing  the  material  in 
greater  depth  on  a second  pass.  Allowing  the  students  to  ask 
questions,  and  the  tutorial  strategy  of  relating  new  material  to  the 
student  s previous  knowledge  contributed  a smaller  amount  to  the 
greater  effectiveness  of  Tutor-SCHOLAR. 

The  fact  that  SCHOLAR  can  be  used  to  test  particular  aspects  of 
teaching  methods  makes  it  potentially  a valuable  tool  for 
educational  research.  The  possibility  of  trying  out  single 
modifications  in  teaching  strategy  to  see  their  effects  on  student's 
learning  rate  is  unique.  Human  teachers  of  course  can  make  such 
modi! ications  in  their  own  teaching  strategies,  but  there  is  no  way 
to  control  all  the  other  factors  that  might  vary  as  they  change 
strategy.  However,  any  specific  version  of  SCHOLAR  is  a fixed 
system,  and  so  an  unbiased  comparison  can  be  made  using  any  number 
of  subjects.  In  this  way  the  accumulation  of  systematic  knowledge 
about  teaching  methods  can  occur. 
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TUTORING  DIFFERENT  TYPES  OF  KNOWLEDGE 

Cne  or  the  limitations  of  the  original  SCHOLAR  was  that  it  was 
restricted  to  teaching  verbal  facts  about  geography.  The  SCHOLAR 
system  itself  has  been  extended  to  encompass  two  other  kinds  of 
knowledge:  visual  knowledge  about  maps  in  tm.  geography  domain,  and 

procedural  knowledge  about  how  to  use  a computer  text-editing  system 
called  NLS  (12).  A related  program  called  SOPHIE  (13)  tutors  the 
diagnostic  skills  needed  in  electronic  troubleshooting.  In  this 
section  we  will  try  to  show  the  generality  of  this  approach  to  CAI , 
and  also  some  of  the  specific  adaptions  that  have  occurred  in 
applying  it  to  different  domains  of  knowledge. 

Tutoring  Visual  Knowledge 

In  order  to  explore  the  tutoring  of  visual  information  in  an 
integrated  manner  with  verbal  information,  we  developed  a 
Map-SCHOLAR  System  (14).  The  system  can  discuss  with  the  student 
different  maps  that  change  dynamically  according  to  the  context  of 
the  discussion.  To  do  this  a graphic  structure  was  created  which 
parallels  the  structure  in  the  semantic  network.  The  elements  in 
the  map  display  therefore  can  be  referred  to  either  by  their  name, 
or  by  pointing  to  them,  or  both.  Map-SCHOLAR  both  asks  and  answers 
map-related  questions  and  provides  relevant  map  information  when  the 
student  makes  a mistake.  It  has  all  the  capabilities  of 
Tutor-SCHOLAR  with  the  addition  of  the  map  capabilities. 


Bolt  Beranek  and  Newman  Inc. 
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Figure  3 illustrates  some  of  the  variety  of  interactions 


possible  with  Map-SCHOLAR.  The  first  panel  illustrates  how 


Map-SCHOLAR  asks  map-related  questions,  e.aluates  the  student's 


answers,  and  corrects  his  mistakes.  ' .-st,  the  dots  indicating  the 


location  of  the  cities  appear  unlabelled  on  the  map  of  Brazil  and 


start  blinking.  Then  SCHOLAR  asks  the  student  to  name  the  blinking 


cities.  When  he  responds,  it  tells  him  which  answers  were  correct 


and  prints  their  names  on  the  map.  Because  he  erroneously  gave  Lima 


as  an  answer,  Map-SCHOLAR  pointed  out  that  Lima  is  in  Peru  and 


showed  where  Lima  is  on  the  map.  Thus  Map-Scholar  gives  both  verbal 


and  visual  feedback  to  help  the  student  learn  the  material. 


The  second  panel  shows  how  context  affects  the  map  display. 


The  topic  is  the  Amazon,  which  appears  in  this  context  on  the  map  of 


Brazil,  the  overall  topic.  The  question  asks  the  student  to  point 


at  the  cities  along  the  Amazon  in  the  order  named.  The  dots  for  the 


three  cities  first  appear  unlabellei,  and  only  after  the  student  has 


answered  do  the  labels  appear.  Unlike  the  first  map,  this  map  shows 


only  the  cities  on  the  Amazon  and  their  relation  to  the  Amazon. 


There  is  also  expansion  of  detail  so  that  a city  like  Belem,  which 
was  not  important  enough  to  be  included  among  the  cities  of  Brazil, 


is  included  with  the  subtopic  of  the  Amazon.  Because  the  map 


changes  dynamically  as  the  context  changes  and  as  the  student 


interacts  with  the  system,  the  students  attention  is  focused  on  the 


relevant  information  exclusively  and  questions  can  be  posed  in  a 


visual  form  not  possible  with  a static  map  display, 


I 


I 

I* 
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NOW  LET'S  SEE  WHAT  YOU  REMEME 
PLEASE  NAME  THE  PRINCIPAL  CIT 
ON  THE  DISPLAY  SCPvEEN. 

*MANAOS  SAO  PAULO  AND  LIMA* 

MANAOS  AND  SAO  PAULO  ARE  CORR 
THE  PRINCIPAL  CITIES  OF  BRAZI 

RIO  DE  JANEIRO  IS  IN  SOUTHEAS 
IT  IS  ON  GUANABARA  BAY  AND  TH 

BRASILIA  IS  IN  CENTRAL  BRAZIL 
IT  IS  FAR  FROM  THE  OCEAN. 


BRASILIA 


ORTO  ALCCRE 


Fio.  3 Interactions  with  ’*ap-SClfOLAP. 


LJ 

LJ 

U 


PLEASE  POINT  TO  THE  FOLLOWING  CITIES  OF  THE  AMAZON 
IN  THE  ORDER  IN  WHICH  THEY  APPEAR  BELOW: 

MANAOS,  BELEM,  AND  IQUITOS 

DON  T FORGET  TO  LET  ME  KNOT  WHEN  YOU  ARE  FINISHFD 

25SJEK T”E  A" ASTERISK  * 


** 


MANAOS  IS  CORRECT. 
THE  CITIES  OF  THE 


YOU  DID  NOT  POINT  AT  BELEM  AND  IQUITOS 
AMAZON  ARE  BLINKING  ON  THE  SCREEN. 


CORRECTLY . 


FiCT*  3 Jntoractions  with  Map-SCHOLAR  (cont) 


LET’S  BEGIN  BY  TALKING  ABOUT  THE  TOPOGRAPHY  OF  VENEZUFLA 
THE  PRINCIPAL  LAKE  OF  VENEZUELA  IS  BLINKING,  PLEASE  NAME " IT . 

*1  DON'T  KNOW* 

THE  PRINCIPAL  LAKE  OF  VENEZUELA  IS  BLINKING  ON  THE  SCREEN. 

LAKE  MARACAIBO  IS  IN  NORTHWESTERN  VENEZUELA 
THE  PRINCIPAL  MINERAL  IS  OIL. 

*IS  THIS  COUNTRY  COLOMBIA?* 

YES,  THAT’S  COLOMBIA 


* WHERE  IS  THE  CAPITAL  OF  VENEZUELA?* 
CARACAS  IS  BLINKING. 


I 


4 A 


U 

u 


r 


FiG.  3 Interactions  with  Man-SPHOLAR  (cont) 
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The  third  panel  shows  the  system's  potential  for  tutorial 
interaction.  When  the  student  didn't  know  about  Lake  Maracaibo, 
Map-SCHOLAR  showed  it  on  the  screen  and  added  the  related  verbal 
information  about  the  oil  there.  The  two  questions  by  the  student 
illustrate  some  of  the  variety  of  questions  the  student  can  ask 
Map-SCHOLAR  to  clarify  or  amplify  the  information  given  (14).  The 
first  question  involves  both  pointing  at  and  naming  Colombia  to  ask 
if  that  is  the  country  near  Lake  Maracaibo.  The  second  question 
shows  another  kind  of  map  question.  The  subject  verbally  asks  where 
the  capital  of  Venezuela  is,  perhaps  to  find  out  how  far  away  it  is. 
SCHOLAR  figures  out  semantically  that  the  capital  is  Caracas,  and 
then  visually  shows  ’/here  Caracas  is  by  blinking  it  (it  is  the 
double  square).  Theje  examples  illustrate  some  of  the  power  for 
tutorial  interaction  that  can  be  obtained  by  a close  integration 
between  semantic  and  visual  knowledge. 

There  is  now  substantial  evidence  in  the  psychological 
literature  (15)  that  people  remember  imageable  material  better  than 
non-imageable  material.  Furthermore,  if  they  us^  image  strategies 
for  remembering  (16),  they  remember  any  given  information  better 
than  if  they  use  other  strategies.  Hence  the  fact  that  information 
is  presented  visually  should  make  it  more  memorable. 

As  Collins  & Quillian  and  Norman  (10)  argue,  the  be:'  way  to 
learn  something  is  to  relate  it  as  much  as  possible  to  whatever 
information  one  already  knows.  Hence,  even  non-visual  information 
like  the  fact  that  the  Manaos  has  a tropical  climate,  will  be 
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learned  better,  if  one  can  see  where  Manaos  is.  This  is  true  for 
two  reasons.  First,  because  when  Manaos  is  located  visually,  it 
will  be  remembered  better,  and  so  facts  that  are  related  to  it  will 
also  be  remembered  better.  Second,  if  a student  sees  that  Manaos  is 
on  the  Amazon,  then  its  climate  can  be  related  to  any  knowledge 
about  the  climate  of  the  Amazon.  Thus,  information  that  seems  not 
to  depend  on  imagery  should  be  remembered  better  in  a visual 
context.  For  these  reasons  we  expect  the  map  facility,  to  have  a 
substantial  impact  on  how  much  students  learn. 

We  have  conducted  an  experimental  evaluation  of  the  map  system 
using  the  comparative  method  described  earlier  (11).  The  test 
compared  student's  learning  with  Tutor-SCHOLAR  using  the  map  system 
vs.  a labeled  map  's.  an  unlabelled  map.  The  experiment  found  an 
advantage  of  the  map  system  over  either  of  the  static  maps.  We  are 
using  a technique  called  "backtrace  analysis",  which  involves 
comparing  the  specific  information  each  student  learned  with  how 
that  information  was  discussed,  in  order  to  pinpoint  what  aspects  of 
the  map  system  led  to  better  learning  by  the  students. 

JnS.Qrlng  Procedural  Knowledge 

NLS-SCHOLAR  (17)  is  a prototype  system  to  teach  computer-naive 
people  how  to  use  the  powerful  NLS  text-editing  system  (12).  This 
teaching  is  accomplished  by  presenting  a sequence  of  lessons. 
During  each  lesson  the  student  may  interact  with  the  system  by 
asking  and  answering  questions,  performing  tasks  which  are  posed  by 
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the  system,  and  performing  tasks  of  his  own  choosing.  Tasks  are 
executed  on  an  actual  NLS  system.  Those  tasks  which  have  been  posed 
are  evaluated  by  the  system,  and  the  student  is  given  encouragement, 
advice,  and  assistance. 

NLS-SCHOLAR  has  been  designed  with  the  belief  that  procedural 
knowledge  is  best  learned  'by  doing'.  It  is  an  example  (18)  of  a 
new  kind  of  CAI  system  that  integrates  systematic  teaching  with 
actual  practice,  i.e.,  one  which  can  keep  a student  under 
"intelligent”  tutorial  supervision  while  allowing  him  to  try  out 
what  he  learns  on  the  very  system  he  is  learning  about.  Thus  the 
system  "knows”  what  the  student  is  doing  and  can  point  out  his 

mistakes,  give  specific  help,  show  him  how  to  do  things  and  even  do 
them  for  him. 

NLS-SCHOLAR  delivers  a series  of  lessons  designed  for  gradual 
understanding  of  NLS  concepts  and  commands.  Within  these  lessons, 
the  system  pauses  to  ask  the  student  questions  and  to  propose 
editing  tasks  for  him  to  perform  using  NLS.  A student's  responses 
to  questions  and  his  performance  of  tasks  are  evaluated  by  the 
system  and  if  he  makes  an  error,  the  nature  of  his  mistake  is 
pointed  out  and  appropriate  action  is  taken.  For  example,  if  a 
question  is  answered  unsatisfactorily,  NLS-SCHOLAR  proposes  another 
question  of  the  same  kind.  If  a task  is  performed  incorrectly, 
depending  on  the  magnitude  of  the  error,  NLS-SCHOLAR  either  resets 
it  for  the  student  to  try  again,  or  asks  him  to  proceed  and  try  to 
fix  his  mistake,  aided  by  the  information  NLS-SCHOLAR  provides. 
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The  user  can  formulate  requests  in  relatively  unconstrained 
English.  The  requests  can  be  questions  about  NLS  concepts  or  about 
the  state  of  his  work,  requests  for  help  in  doing  a task,  or  even 
NLS  commands  expressed  in  English.  The  system  is  "aware”  of  what 
the  user  is  currently  doing  so  that  his  requests  for  help  can  be 
answered  within  the  context  of  the  problem  he  is  working  on.  Thus 
NLS-SCHOLAR  not  only  tells  him  "The  general  procedure  is..."  but 
also  "In  your  case,  what  you  should  do  is...". 

NLS-SCHOLAR  has  the  ability  to  use  the  NLS  file  a student  is 
currently  working  on  to  show  him  how  to  perform  editing  actions. 
This  gives  the  system  much  of  the  flavor  of  a human  tutor,  as  if  he 
were  taking  the  student's  place  at  the  terminal  and  saying  "Watch  me 
do  it  for  you". 

The  flavor  of  NLS-SCHOLAR  is  best  conveyed  by  an  annotated 
protocol,  shown  in  Fig.  4,  which  was  obtained  on-line  using  a 
version  of  the  system.  In  the  course  of  a lesson,  students  learn 
how  to  change  the  contents  of  a menu  by  performing  editing 
operations.  The  protocol  starts  at  a point  well  along  in  the 
student's  learning  of  NLS  — he  has  been  told  about  NLS  files,  how 
to  load  them,  print  them,  delete  and  insert  statements,  etc.  He  is 
about  to  be  taught  how  to  use  the  Substitute  command  to  effect  a 
change  in  the  menu.  Notice  that  what  the  student  actually  typed  is 
underlined  and  that  our  annotations  appear  in  italics. 

The  capabilities  of  NLS-SCHOLAR  allow  students  to  learn  from 
explanation,  learn  by  doing,  and  learn  by  asking  questions.  Their 
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BREAKFAST  MEN l 


JUICE 

1 A ORANGE 

IB  GRAPEFRUIT 

CEREAL 

2A  OATMEAL 

2 A 1 WITH  RAISINS 
2 B CREAM  OF  WHEAT 
2 C CORN  FLAKES 
EGGS 

3A  SCRAMBLED 
3B  FRIED 

3B1  SUNNY-SIDE-UP 
3B2  OVER-EASY 
3C  BOILED 
BEVERAGE 

UA  HOT  CHOCOLATE 
4B  TEA 

HB1  WITH  LEMON 
*»B2  WITH  SUGAR  AND  CREAM 
*IC  COFFEE 


SUBSTITUTE  WORD  IN  STATEMENT 


Before  we  end  this  lesson,  I d like  to  show  you  how  to  change  the 
content  of  statements  which  have  already  been  inserted  in  your  file. 


You've  learned  how  to  correct  errors  by  deleting  an  entire  statement 
and  inserting  anew  one  in  its  place.  The  Substitute  command  nay  be 
used  to  change  just  a word  or  even  a single  character  at  a time.  For 

u;™Cn%fchOW  y.°U  C3n  Chann°  3 Vord  in  statement  2A 1 so  that  it 
reads  WITH  DATES  instead  of  WITH  RAISINS. 


EDIT  C:  (Su  )bstitute  C:  (W  )ord  in 
) ( <CR> ) ; 

<New  WORD>  T:  ( DATES) ( <CR> ) ; 

<01d  WORD>  T:  ( RAISINS) ( <CR> ) ; 
Finished?  Y/H:  (Y)  OK:  (<CR>);  ’ 
Substitutions  made:  1 
EDIT  C:  (Do  )ne  OK:  (<CJ>) ; 


(S  )tatement  at  A:  (2A1 


NLS- SCHOLAR  performs  a command  for  the  student  much  as  a 
human  tutor  could  by  leaning  over  the  student  and  typing 
the  command  on  the  student's  terminal.  Parentheses'  enclose 
uhat  the  ’tutor"  actually  "types",  separating  it  from  the 
echoes  and  responses  printed  by  i.'LS. 

<CR>  is  just  a symbol  to  denote  that  the  "tutor"  depressed 
the  return  key.  The  semicolon  is  llLS’s  echo  to  <CH>. 


Fiq. 


Dialoque  with  TTT.S-SCHOLAR 
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L t wassm m 


You  can  sec  that  I've  specified  that  I want  to  substitute  one  word  in 
statement  2A1  for  another;  the  new  word  DATES  is  to  be 
substituted  for  the  old  word  RAISINS. 

(There  is  no  restriction  that  the  items  substituted  for 
one  another  be  of  the  same  length.) 

To  do  this  I typed  only  *Su  W S 2A1  <CR> ' to  produce  the  first  line;  I 
typed  'DATES  <CR>'  and  'RAISINS  <CR>'  to  produce  the  second  and  third. 
The  remaining  characters  are  all  printed  by  NLS. 

NLS  asks  'Finished?'  since  more  than  one  substitution  can  be  made  at  a 
time . If  I had  typed  'H'  for  no  instead  of  'Y'  for  yes,  it  would  have 
prompted  for  another  <New  UORD>  and  <01d  W0RD>  pair. 

After  I typed  'Y'  NLS  asks  for  further  confirmation  with  OK: 
to  which  I responded  with  a <CR>. 

When  the  command  has  been  executed,  NLS  tells  how  many  substitutions 
have  been  made.  If  there  had  been  two  instances  of  RAIS1IIS  in 
statement  2A1,  both  would  have  been  changed  to  read  DATES. 

Statement  2A1  is  now  the  current  statfcment.  Please  print  it. 


TASK  14 1 

EDIT  C:  /_ 
rr>WITH  DAT 
EDIT  C:  !_ 

2A  OATMEAL 


EDIT  C: 

2A1  WITH  D/^ES 
EDIT  C:  Done  OK: 


Student  typing  to  underlined. 

The  student  confuses  the  \ command,  which 
prints  the  current  statement,  witn  the  / command 
that  shows  the  current  position  of  the  control 
marker.  Kcalir.ing  his  mistake,  he  fumbles  with 
the  other  one-character  commands  he  can 
remember.  First  he  tries  t which  prints  the 
previous  statement  (Z<\),  and  makes  it  become 
the  current  statement. 

tiext  the  student  depresses  the  LINE  FEF.P 
key  which  echoes  as  S,  giving  him  the  next 
statement  (ZAl),  making  it  current. 


You  printed  too  many  statements. 
This  is  what  I wanted  you  to  print: 
2A1  WITH  DATES 


The  task  is  evaluated. 


Please  redo  just  the  printing 
part  of  this  task.  Your  file  is  unchanged. 

NLS-SCUQLAP.  is  really  determined  to  get  the  student  to  review 
how  to  print  the  current  statement. 


’ig.  4 Oialoque  with  *JLS-SC!!OT,AF  (cont) 
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continuation  of  TASK  <4  1 

EDIT  C:  »Uhat  command  prints  the  current  statement?* 

THE  USER  PRINTS  THE  CURRENT  STATEMENT  USING  THE 
\ COMMAND. 

Failing  to  remember  the  appropriate  one-character  command 
the  student  gives  up  and  asks  for  help . 

Questions  can  be  asked  anytime  it  is  the  student's  turn  to  type. 
HLS- SCHOLAR  answers  this  question  by  retrieving  information  from 
its  oun  internal , descriptive  data  base. 

continuation  of  TASK  <11 
EDIT  C: 

2A1  WITH  DATES 
£DIT  C:  Done  OK:  ; 

QUESTIONS? 


The  student  chooses  to  ask  no  questions  at  this  time. 

7 ry  the  Substitute  coninand  yourself  by  substituting  the  word 
OAT  for  COHN  in  the  statement  containing  "COIIll  FLAKES". 


D 


TASK  <12 
EDIT  C:  “WI- 


THE STATEMENT  CONTAINING  "CORN"  IS: 

2C  CORN  FLAKES 

The  student  Hants  to  knou  uhat  the  correct  statement  number  is 
for  him  to  type.  The  system  obtains  the  ansucr  by  actually 
looking  into  the  student's  vork  space  --  the  ansucr  is  not  stored 
(it  cannot  be.')  in  the  internal  data  base. 

The  sys  t cm  interprets  correctly  most  of  the  common 
paraphrases  of  this  type  of  question , e.g. 

"What  is  the  statement  containing  ..." 

"What  statement  contains  ..." 

"Where  is  ...  ",  etc. 


Fig.  4 Dialogue  with  NLS-SOFOLAP.  (cont) 
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continuation  of  TASK  *12 

EDIT  C:  Substitute  C:  Word  in  C:  Statement  at  A:  £C ; 
<llew  WORD>  T:  COJItl ; 

<01d  WORD>  T:  OAT; 

Finished?  Y/ll:  Y OK 
Substitutions  made:  0 
EDIT  C:  Done  0K:__; 


The  student  switches 
so  no  change  ic  made. 

You  should  have  altered  the  file 
so  that  this  part  of  it 
2B  CREAII  OF  UliFAT 
2C  CORIi  FLAKES 
3 • • • 


would  look  like  thin: 

213  CIIKAIl  OF  WHEAT 
2C  OAT  FLAKES 
3 • • • 

But  you  did  not  alter  the  file. 

Please  try  to  repair  your 
mistake.  The  file  is  as  you  left  it. 


around  the  new  and  the  old  words. 


HLS-SCHOLAH  chows  the  student 
what  the  result  of  this  comma:. d 
should  have  been  if  he  had 
performed  the  task  correctly 
as  opposed  to  the  result  which 
he  obtained.  With  this 
information,  the  student 
can  "debug"  hi:;  performance 
of  the  task. 


continuation  if  TASK  *12 


EDIT  C:  *How  does  one  do  this?*  The  student  is  lost  and 

asks  for  help. 

THE  USER  TYPES  'SUBSTITUTE',  FOLLOWED  BY  'WORD', 

'STATEMENT',  '2C',  <CR> , 'OAT',  <CR>, 

'CORN',  <CR> , 'Y',  <CR> , 'DONE'.,  AND  <CR>. 

NLS-SCHOLAR  c an  show  him  what 
to  do,  as  wel1  as . . . 

continuation  of  TASK  *12 

EDIT  C:  »PJ,?ase  dp  it  for  Jad  ...  actually  do  it  for  him! 


EDIT  C:  (Substitute  ) C:  (Word  ) in  C:  (Statement  ) at  A:  (2C)(<CR>); 
<New  WORD>  T:  (OAT)(<CR>) ; 

<01d  WORD>  T:  (CORN ) (<CR> ) ; 

Finished?  Y/N:  (Y)  OK:  (<CR>); 

Substitutions  made:  1' 

EDIT  C:  (Done  ) OK:  (<CH>)  ; 


Fig.  4 Dialogue  with  NLS-SCHOLAR  (cont) 


28 


BBN  Report  No.  3181 


Bolt  Beranek  and  Newman  Inc 


tight  integration  within  a working  environment  makes  NLS-SCHOLAR 
powerful  tutorial  CAI  system. 


Tutoring  Diagnostic  Skills 

SOPHIE  (13)  reflects  an  attempt  to  extend  Carbonell's  notion  oJ 
mixed-initiative  CAI  for  the  purpose  of  encouraging  a wider  range  of 
student  initiatives.  Unlike  previous  tutorial  systems  which  attempt 
to  mimic  the  roles  of  a human  teacher,  SOPHIE  tries  .o  create  a 
'’reactive”  environment  in  which  the  student  learns  by  trying  out  his 
ideas  rather  than  by  instruction.  To  this  end,  SOPHIE  incorporates 
a ’’strong"  model  of  its  knowledge  domain  along  with  numerous 
heuristic  strategies  for  answering  a student's  questions,  providing 
him  with  critiques  of  his  current  solution  paths,  and  generating 
alternative  theories  to  his  current  hypotheses.  In  essence,  SOPHIE 
enables  a student  to  have  a one-to-one  relationship  with  an  "expert" 
who  helps  the  student  create,  experiment  with,  and  debug  his  own 
ideas . 

Sophie's  expertise  is  derived  from  an  efficient  and  powerful 
inferencing  scheme  that  uses  multiple  representations  of  knowledge 
including  (a)  simulation  models  of  the  domain  (b)  procedural 
specialists  which  contain  logical  skills  and  heuristic  strategies 
for  using  these  models,  and  (c)  semantic  nets  for  encoding 
time-invariant  factual  knowledge.  The  power  and  generality  of 
SOPHIE  stems,  in  part,  from  the  synergism  obtained  by  focusing  the 
diverse  capabilities  of  the  procedural  specialists  on  the 
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"intelligent"  manipulation,  execution,  and  interpretation  of  i*s 
simulation  models. 


In  the  basic  scenario,  SOPHIE  acts  as  an  electronics  lab 
instructor  who  helps  the  student  transform  his  classroom  knowledge 
of  electronics  into  an  experiential,  intuitive  knowledge  of  its 
meaning  and  application.  It  does  this  .y  interacting  with  the 
student  while  he  is  debugging  a malfunctioning  piece  of  equipment 
09).  The  student  can  perform  any  sequence  of  measurements,  ask 
either  specific  questions  about  the  implications  of  these 
measurements  for  more  general  hypothetical  questions,  and  even  ask 
for  advice  about  what  to  consider  next,  given  what  he  has  discovered 
thus  far.  At  any  time  SOPHIE  may  encourage  the  student  to  make  a 
guess  as  to  what  he  thinks  might  be  wrong  given  the  measurements  he 
has  made  thus  far.  If  he  does,  SOPHIE  will  evaluate  his  hypothesis 
by  taking  into  consideration  all  the  information  he  should  have  been 
able  to  derive  from  his  current  set  of  measurements.  If  any  of  this 
information  is  logically  contradicted  by  the  hypothesis,  SOPHIE 
identifies  and  explains  these  contradictions.  Likewise  SOPHIE  can 
Judge  the  merits  of  any  particular  measurement  with  respect  to  the 
prior  sequence  of  measurements  he  has  made.  For  example,  his  new 
measurement  may  be  logically  redundant  in  the  sense  that  no  new 
information  can  possibly  be  derived  from  it  (an  extremely  „mplex 
task  to  determine).  SOPHIE  can  also  decide  if  this  measurement 
p forms  a reasonable  split  of  the  hypothesis  space  of  possible 
faults  which  have  not  yet  been  ruled  out  by  prior  measurements. 
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It  should  be  noted  that  the  scenario  contains  quite  a variety 
of  logical  tasks  (i.e.,  hypothesis  evaluation,  hypothesis  formation, 
redundancy  checking,  hypothetical  question  answering)  each  one  of 
which  requires  a substantial  amount  of  deep  logical  inferencing. 
One  of  the  basic  challenges  in  constructing  SOPHIE  was  creating  an 
inference  system  which  could  perform  this  wide  range  of  tasks 
efficiently  (so  that  it  could  be  used  in  real  time)  and  at  the  same 
time  have  it  be  robust  in  the  sense  of  handling  all  realistic 
queries . 

Because  SOPHIE  was  designed  as  an  environment  in  which  students 
could  create  and  articulate  ideas,  it  was  necessary  to  have  a 
powerful  natural  language  processor  to  communicate  with  students.  A 
student  will  become  frustrated  if  he  has  to  try  several  ways  of 
expressing  an  idea  to  get  a response.  In  addition  he  will  become 
bored  if  there  is  a long  delay  (say  10  secs)  before  the  system 
replies.  And  because  students  begin  to  assume  the  system  shares 
their  "world-view",  SOPHIE  must  cope  with  contextually-dependent 
references,  deletions,  and  ellipses.  SOPHIE 's  natural  language 
processor  is  based  upon  a "semantic  grammar"  technique,  in  which 
concepts  like  "measurement"  or  "circuit  element"  trigger 
expectations  about  what  things  should  appear  in  the  student's  input. 
SOPHIE  has  demonstrated  that  natural  language  processing  has 
advanced  far  enough  to  deal  with  these  three  kinds  of  difficulties 
well  enough  to  build  friendly,  but  sophisticated  tutorial  systems. 
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THE  FUTURE  OF  INTELLIGENT  CAI 

The  thrust  of  this  paper  has  been  to  show  what  ..ind  of 
capabilities  are  now  available  for  building  genuinely  intelligent 
CAI  systems.  The  domain  of  such  systems  is  virtually  unlimited;  it 
is  not  restricted,  for  example,  to  drill  and  practice  or 
mathematics.  The  language  capabilities  of  current  systems  are  not 
equal  to  those  of  a human,  nor  will  they  be  in  the  foreseeable 
future,  but  they  are  good  enough  to  sustain  practical  systems. 

The  Plato  system  (6)  has  shown  that  it  is  possible  to  ^uild 
both  interesting  and  cost-effective  CAI  systems  in  a time-shared 
computer  environment.  They  have  accomplished  this  by  using  a 
variety  of  teaching  techniques:  the  Socratic  method,  generative  CAI, 
games  and  simulations,  programmed  instruction,  etc.  Intelligent  CAI 
is  an  attempt  to  go  beyond  the  technology  in  the  Plato  system  to 
explore  how  to  build  greater  intelligence  into  tutorial  systems, 
while  at  the  same  time  utilizing  many  of  the  educational  techniques 
employed  so  successfully  in  Plato. 

Intelligent  CAI  systems  are  now  both  costly  to  build  (above 
$100,000)  a.id  to  use  (about  $10-$20  per  hour).  But,  the  cost  of 
computing  continues  to  decrease  while  teacher's  salaries  are  rising. 
Hence  the  cost  of  running  such  systems  should  be  competitive  in 
comparison  to  the  cost  of  human  tutoring  within  a short  time, 
especially  where  there  are  few  skilled  teachers  available,  as  with 
teaching  computer  text-editing.  The  effective  cost  of  building  such 
systems  depends  on  how  much  they  are  used.  If  they  are  used 
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heavily,  then  the  large  cost  of  building  them  will  be  worth  the 
investment;  otherwise  not.  It  Is  at  least  possible  that  one  of  the 
current  systems  will  be  used  enough  to  justify  the  development 
expense,  though  they  were  built  only  as  prototype  systems.  The  test 
though  will  be  the  development  of  such  a system  for  a sohool  setting 
where  large  numbers  of  people  are  being  taught. 


The  payoff  m Intelligent  CAI  oomes  from  personalizing  the 
learning  process.  Personalization  is  effective  in  many  wa. s : by 
forcing  the  student  to  participate  in  learning;  by  teacning  at  the 
level  of  his  individual  knowledge;  by  providing  a setting  where  the 
student  can  try  out  his  own  ideas  and  make  mistakes;  by  freeing  the 


student  from  peer  pressure;  by  addressing  the 
confusions,  etc.  These  advantages  make  it 

intelligent  CAI  a serious  trial. 


student's  individual 
worthwhile  to  give 
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